Previously, a system and method for producing synthetic single or plural word messages was developed by Bruce Baker et al. and is disclosed in U.S. Pat. No. 4,661,916 to Baker et al. (the Baker '916 patent), issued on Apr. 28, 1987, the entire contents of which are hereby incorporated herein by reference. The system was directed to a linguistic coding system and keyboard for use by people with cognitive and/or physical impairments. The coding system and associated keyboard was used to store and access messages, which included plural word messages, sentences, phrases, full names, letters, numbers, functions, or any combination thereof.
In such a system, the keyboard was coupled to a computer device, or was alternately part of the stand-alone entity which included a microprocessor, memory and display. The memory stored the messages for selective retrieval by the keyboard. The messages retrieved from the keyboard were then fed to a voice synthesizer, for example, which converted them through a loudspeaker to produce audible spoken messages. On this keyboard, associated with each of a plurality of keys, were polysemous (many-meaning) symbols, also known as icons. By designating selected ones of the keys and their associated symbols or icons, selected stored messages or plural word messages (including but not limited to words, phrases and sentences) were accessed from the memory and then subsequently output.
With the system described in the Baker '916 patent, messages stored in the memory could be retrieved by activating a combination of symbol keys and other keys to vary the context of the polysemous symbols. Thus, a plurality of sentences could be selectively generated as a function of polysemous symbols in combination with other polysemous symbols. This allowed a user the ability to access thousands of words or messages based upon as little as one, two, or three keystrokes. Further, with symbols being polysemous, thousands of symbol sequences could be generated with only a small number of keys on a keyboard. Based upon ease of use of the system, the polysemous icons or symbols utilized, and the easily memorized symbol sequence combinations, such a system became ideal for many mentally and physically challenged users for whom spelling and typing, as well as speech itself, was extremely difficult.
The system of the Baker '916 patent allowed for an operator to go directly from thought to speech. This was possible because each key of the keyboard bore a central image or symbol which was polysemous and illustrated an important aspect of life and/or linguistic function. The keyboards could be varied depending on the intellectual level of the intended operator. Therefore, each keyboard could in itself be a language which was designed for or with a specific user.
Each of the polysemous symbols was developed to be rich in associations and in combination, signal sentence or message ideas in the operator's memory. This enabled the generation of plural word or whole sentence messages by the activation of only a limited number of keys. The device allowed for the generation of many sentences or phrases and a large core vocabulary which could be easily retrieved from memory because of the ease with which the polysemous symbols on the keys portrayed the production of whole thoughts.
In the aforementioned system of the Baker '916 patent, the spatial configuration of the symbols on a given keyboard remained constant. Sequences of icons in fixed places were consistent, allowing messages to be reliably produced with the same sequence each time. This constant mapping supported the learning of motor patterns associated with icon sequences. As such sequences were learned, the user could establish motor programs that allow sequences to be produced quickly and accurately in the same way a touch typist efficiently spelled many words or a musician played an instrument.
The aforementioned Baker '916 patent provided an excellent means of accessing high frequency “core” vocabulary words using sequenced polysemous symbols. However, the system of the Baker '916 patent only provided limited access to the relatively large set of low frequency “fringe” vocabulary words that would only be used periodically.
A subsequent design that provided for a way to easily access fringe vocabulary utilizing non-polysemous symbols on dynamic graphical screens was disclosed in U.S. Pat. No. 5,920,303 to Baker et al. (the Baker '303 patent), issued Jul. 6, 1999, the entire contents of which is hereby incorporated herein by reference. In the system of the aforementioned Baker '303 patent, less than all of a plurality of displayed symbols on a keyboard were polysemous symbols that could be used sequentially for producing core vocabulary with the advantages of the Baker '916 patent design, and less than all of the plurality of displayed symbols on the keyboard were non-polysemous symbols for accessing fringe vocabulary. In the system of the Baker '303 patent, the less than all of a plurality of keys on the displayed keyboard including non-polysemous symbols for accessing fringe vocabulary were dynamically redefined in response to sequentially selected polysemous or non-polysemous symbols, such that both the stored message accessed by actuation of a key and the non-polysemous symbol displayed on the key were simultaneously dynamically redefined. These dynamic characteristics produced a dynamically redefined keyboard for accessing fringe vocabulary.
The aforementioned Baker '916 and Baker '303 patents provided for visually based approaches to polysemous symbolic representation, including pictorial illustrations and alpha characters. In various embodiments, these symbols have been printed on physical displays and rendered as virtual objects on computer displays.
Contemporary embodiments exist on touch screen computers. However visually based symbolic representation is presented, and as such, visual processing skills are needed to access these symbols. The use of visually based symbolic representation assumes the user has adequate vision to differentiate one symbol from the next.
Visually based polysemous and non-polysemous symbols have helped many individuals with significant speech and multiple impairments (SSMI) communicate with natural language and generate spontaneous novel utterances. However, a subset of this population does not have adequate vision to process pictorial illustrations and alpha characters. Historically, these users have relied on either memorization of button locations with no accessible symbolic representation, or auditory scanning to access language content, if a speech output system is used. Some individuals with low vision may have the option to use systems with a relatively small number of enlarged symbols and a smaller vocabulary set. Many of these individuals have had to rely on home-made tactile systems with no speech output.
If a user relies exclusively on memorization of button locations on a touch screen display or key locations on a keyboard, there is no symbolic representation associated with a location to convey meaning before the location is activated. An auditory prompting feature may be used. Auditory prompting occurs when an AAC system speaks the name of a symbol upon the first activation as a prompt, so that a second activation of the same icon is required to select it. However, the use of auditory prompting automatically doubles the number of activations that are necessary to generate any given utterance.
If a user relies on auditory scanning, communication is slower and more cognitively demanding than with direct selection techniques. The user must wait for the system to scan through a series of options in sequence, and select each target symbol in turn. Depending on the scanning method, a user may need to make as many (such as three for example) correctly timed selections to activate one symbol, and must further retain the desired utterance in memory throughout this process.
Others in the field of augmentative and alternative communications (AAC) have attempted to provide access to words, phrases, and sentences through alphabet based methods (spelling, word prediction, and orthographic word selection), and through single-meaning pictures on dynamic, graphic screens.
A person who is visually impaired or even blind may spell using a touch keyboard. As discussed in the Baker '303 patent, spelling is often slow, difficult, and laborious for people with significant speech and multiple impairments (SSMI), particularly if they have motor impairments that limit their potential for touch typing. If a chorded Braille keyboard is used for spelling, multiple keys must be actuated simultaneously to produce each character, increasing the motoric demands of spelling.
Unfortunately, many individuals with SSMI have persistent difficulty acquiring and using literacy skills. Literacy is particularly challenging for individuals who have significant speech impairments and blindness, and must learn to read and write in Braille.
Word prediction, orthographic word selection, and single meaning pictures are all strategies that are reliant on touch screen graphic displays. In word prediction, a person types one letter of a target word. The computer then presents the user with a list of words beginning with the chosen letter. The user scans the list to determine if the target word is included, then either selects the target word from the list or types the next letter if the target word is not listed.
In a single meaning picture system, an array of single meaning pictures may be presented for the user to choose from. If the size of a user's vocabulary exceeds the small number of picture locations on the screen, more screens may be progressively added. Multiple activations are necessary to navigate between many different screens and select a desired word or formulate an utterance. A user with a large vocabulary may have their vocabulary distributed across many dozen screens. Orthographic word selection functions similarly to a single meaning picture system, except that symbols consist of whole words presented in alpha characters instead of pictures.
Utilizing word prediction, orthographic word selection, or single meaning pictures on a touch screen graphic display is cognitively demanding and requires concentration, as discussed in the Baker '303 patent. These methods present additional challenges to users who are visually impaired or even blind, when they must process the information from a dynamic word prediction menu or a touch screen using other sensory modalities. A user with impaired or even without vision may need to rely on auditory prompting or auditory scanning to access such information.
Known home-made tactile symbol systems tend to be built using manual construction techniques, and tend to include symbols which are single meaning and represent low frequency fringe vocabulary. These symbol systems may be implemented as manual communication boards with no speech output capabilities, or mounted to digitized voice output devices with limited storage capacities. Using home-made tactile symbols is cognitively demanding for users, particularly when no speech output is available.
Such known systems suffer from the known limitations of many communication systems that use single meaning pictures. Additionally, most of these systems do not include voice output, and the user is burdened with the need to physically manipulate a real object for every symbol in the language system.
After reviewing clinical practice with these patients, existing technologies were examined to determine if there was a gap that explained why numerous sighted patients receive speech generating devices while most patients with visual impairments do not. In a review of user interface systems on twenty one speech generating devices from seven manufacturers, an abundance of visual language representation and auditory scanning options were found. Tactile interface options across devices and manufacturers were limited to keyguards that help users isolate targets, spelling and typing systems, and two devices which could potentially support a tactile symbol set created by consumers. Thus, the needs of these patients were not being met, in part because the available technology was not consistent with clinical practice.
Fabricating home-made tactile symbol systems further is time and labor intensive, and cannot keep pace with the development of linguistic skills. For example, a normally developing three year old has an expressive vocabulary of about 1,000 words. A 5 cm×5 cm tactile symbol to represent each of these words would require a display with at least 2.5 m×2 m of usable surface area. An attempt to keep pace with the development of linguistic skills with single meaning tactile symbols quickly produces a very large symbol set that is extremely difficult to manage.